Skip to content

HADOOP-19571. Improve PrometheusMetricsSink#normalizeName performance#7692

Merged
slfan1989 merged 4 commits intoapache:trunkfrom
ivandika3:HADOOP-19571
May 22, 2025
Merged

HADOOP-19571. Improve PrometheusMetricsSink#normalizeName performance#7692
slfan1989 merged 4 commits intoapache:trunkfrom
ivandika3:HADOOP-19571

Conversation

@ivandika3
Copy link
Contributor

@ivandika3 ivandika3 commented May 18, 2025 •
edited by slfan1989
Loading

Description of PR

JIRA: HADOOP-19571. Improve PrometheusMetricsSink#normalizeName performance

This patch is similar from HDDS-13014. We can add a name normalization cache between the Hadoop metrics name and the Prometheus metrics name to prevent expensive regex matchings during the metric normalization conversion.

How was this patch tested?

Simple microbenchmark (see https://issues.apache.org/jira/secure/attachment/13076557/TestPrometheusMetricsSinkPerformance.java)

# With cache

Warming up...
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by org.apache.hadoop.thirdparty.com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper (file:/Users/ivan.andika/.m2/repository/org/apache/hadoop/thirdparty/hadoop-shaded-guava/1.4.0/hadoop-shaded-guava-1.4.0.jar)
WARNING: Please consider reporting this to the maintainers of class org.apache.hadoop.thirdparty.com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release

Running performance test...

Performance Test Results:
Total test cases: 5
Total iterations: 100000
Total operations: 500000
Total time: 46.00 ms
Average time per operation: 0.000 ms

Process finished with exit code 0

# Without cache

Warming up...

Running performance test...

Performance Test Results:
Total test cases: 5
Total iterations: 100000
Total operations: 500000
Total time: 1181.00 ms
Average time per operation: 0.002 ms

@ivandika3 ivandika3 marked this pull request as ready for review May 18, 2025 13:43
@hadoop-yetus

This comment was marked as outdated.

Sign in to view
@hadoop-yetus
Copy link

hadoop-yetus commented May 19, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 1s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 36m 0s trunk passed
+1 💚 compile 15m 53s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 compile 13m 58s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 checkstyle 1m 17s trunk passed
+1 💚 mvnsite 1m 38s trunk passed
+1 💚 javadoc 1m 18s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 56s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 2m 35s trunk passed
+1 💚 shadedclient 37m 15s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 55s the patch passed
+1 💚 compile 15m 2s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javac 15m 2s the patch passed
+1 💚 compile 13m 54s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 javac 13m 54s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 9s the patch passed
+1 💚 mvnsite 1m 36s the patch passed
+1 💚 javadoc 1m 12s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 55s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 2m 43s the patch passed
+1 💚 shadedclient 36m 21s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 15m 24s hadoop-common in the patch passed.
+1 💚 asflicense 1m 4s The patch does not generate ASF License warnings.
203m 28s
Subsystem Report/Notes
Docker ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/2/artifact/out/Dockerfile
GITHUB PR #7692
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 3d6ffc297392 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ba85667
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/2/testReport/
Max. process+thread count 2152 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ivandika3
Copy link
Contributor Author

ivandika3 commented May 19, 2025

@aajisaka Please kindly help take a look when you have time.

@aajisaka aajisaka self-requested a review May 20, 2025 05:19
aajisaka
aajisaka approved these changes May 20, 2025
View reviewed changes
Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ivandika3 ivandika3 mentioned this pull request May 20, 2025
Merged
@slfan1989
Copy link
Contributor

slfan1989 commented May 21, 2025

@ivandika3 Thanks for the contribution! if there are no other comments, I will merge this PR tomorrow.
cc: @aajisaka

@slfan1989
Copy link
Contributor

slfan1989 commented May 22, 2025

@ivandika3 Could we retrigger the build? It seems the current Yetus status is not accurate.

@ivandika3
Copy link
Contributor Author

ivandika3 commented May 22, 2025

@slfan1989 Thanks for checking this, pushed an empty commit to trigger build.

@hadoop-yetus
Copy link

hadoop-yetus commented May 22, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 58s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 39m 0s trunk passed
+1 💚 compile 18m 42s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
-1 ❌ compile 2m 38s /branch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in trunk failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
+1 💚 checkstyle 1m 25s trunk passed
-1 ❌ mvnsite 0m 34s /branch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in trunk failed.
+1 💚 javadoc 1m 26s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 58s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 3m 14s trunk passed
-1 ❌ shadedclient 43m 5s branch has errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 23s /patch-mvninstall-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ compile 0m 23s /patch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 23s /patch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ compile 0m 23s /patch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in the patch failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
-1 ❌ javac 0m 23s /patch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in the patch failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 21s /buildtool-patch-checkstyle-hadoop-common-project_hadoop-common.txt The patch fails to run checkstyle in hadoop-common
-1 ❌ mvnsite 0m 24s /patch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ javadoc 0m 24s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ javadoc 0m 23s /patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt hadoop-common in the patch failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
-1 ❌ spotbugs 0m 23s /patch-spotbugs-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
+1 💚 shadedclient 5m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 24s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
+0 🆗 asflicense 0m 23s ASF License check generated no output?
118m 15s
Subsystem Report/Notes
Docker ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/3/artifact/out/Dockerfile
GITHUB PR #7692
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux a087249a63a0 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ad21951
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/3/testReport/
Max. process+thread count 258 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989
Copy link
Contributor

slfan1989 commented May 22, 2025

@ivandika3 I noticed some errors in the build results, but they are not caused by this PR. Could we sync the latest changes from trunk using 'sync fork'?

image

@hadoop-yetus
Copy link

hadoop-yetus commented May 22, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 22s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 22m 55s trunk passed
+1 💚 compile 8m 27s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 compile 7m 31s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 checkstyle 0m 43s trunk passed
+1 💚 mvnsite 1m 2s trunk passed
+1 💚 javadoc 0m 45s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 35s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 1m 29s trunk passed
+1 💚 shadedclient 22m 10s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 31s the patch passed
+1 💚 compile 9m 10s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javac 9m 10s the patch passed
+1 💚 compile 7m 52s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 javac 7m 52s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 40s the patch passed
+1 💚 mvnsite 1m 0s the patch passed
+1 💚 javadoc 0m 42s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 35s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 1m 33s the patch passed
+1 💚 shadedclient 21m 15s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 13m 24s hadoop-common in the patch passed.
+1 💚 asflicense 0m 41s The patch does not generate ASF License warnings.
124m 34s
Subsystem Report/Notes
Docker ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/5/artifact/out/Dockerfile
GITHUB PR #7692
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 8d295ffb8be0 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f16470a
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/5/testReport/
Max. process+thread count 3151 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7692/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989 slfan1989 merged commit 949292e into apache:trunk May 22, 2025
1 of 3 checks passed
@slfan1989
Copy link
Contributor

slfan1989 commented May 22, 2025

@ivandika3 Thanks for the contribution! Merged into the trunk. @aajisaka Thanks for the review!

@ivandika3
Copy link
Contributor Author

ivandika3 commented May 22, 2025

@aajisaka @slfan1989 Thanks for the reviews.

@ivandika3 ivandika3 deleted the HADOOP-19571 branch May 22, 2025 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aajisaka aajisaka aajisaka approved these changes

@slfan1989 slfan1989 slfan1989 approved these changes

Assignees

No one assigned

Labels

Common trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ivandika3 @hadoop-yetus @slfan1989 @aajisaka